Issues in Statistical Inference

Being critical of using significance tests in empirical research, the Board of Scientific Affairs (BSA) of the American Psychological Association (APA) convened a task force "to elucidate some of the controversial issues surrounding applications of statistics including significance testing and its alternatives; alternative underlying models and data transformation; and newer methods made possible by powerful computers" (BSA; quoted in the report by Wilkinson & Task Force, 1999, p. 594). Guidelines are stipulated in the report for revising the statistical sections of the APA Publication Manual. Some assertions in the report about research methodology are reasonable. An example is the statement, "There are many forms of empirical studies in psychology, including case reports, controlled experiments, quasi-experiments, statistical simulations, surveys, observational studies, and studies of studies (meta-analyses) ... each form of research has its own strengths, weaknesses, and standard of practice" (Wilkinson & Task Force, 1999, p. 594). However, it does not follow that data collected with any two methods are equally unambiguous. At the same time, a method that yields less ambiguous data is methodological superior to one that yields more ambiguous data. That is, despite the assertions made in the report, a case can be made that "some of these [research methods] yield information that is more valuable or credible than others" (Wilkinson & Task Force, 1999, p. 594). It is unfortunate that the report reads more like an advocacy document than an objective assessment of the role of statistics in empirical research. Moreover, non-psychologist readers of the report can be excused for having a low opinion of psychologists' research practice and methodological sophistication. Lest psychologists' methodological competence be misunderstood because of the report, this commentary addresses the following substantive issues: (a) the acceptability of the 'convenience' sample, (b) the inadequacy of the contrast group, (c) the unwarranted belief in the experimenter's expectancy effects, (d) some conceptual difficulties with effect size and statistical power, and (e) the putative dependence of statistical significance on sample size.